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Abstract 



In this paper we study the problem of generation of dependent random variables, known as the 

"coordination capacity" ID ||5], in multiterminal networks. In this model m nodes of the network are 

observing i.i.d. repetitions of X^^\ X^^\..., X^™) distributed according to q{x'-^\ x'^"^''). Given a 

joint distribution q{x''^\ x'^"^\y'^^\ ...,y^"^'>), the final goal of the i*'' node is to construct the i.i.d. 

copies of F^*) after the communication over the network where X'--^\ X'^^\..., X^"^\Y^^\ Y'-^\..., 

' are jointly distributed according to (7(0;^^-', a;*-™-*, y*-^-*, y*-™-*). To do this, the nodes can exchange 

> ■ 

, difficult to solve even for the special case of two nodes. In this paper we prove new inner and outer 

bounds on the achievable rates for networks with two nodes. 

in 
o 



I. Introduction 

Traditionally coding is used in networks to distribute information among the nodes. However in some 
applications (certain) nodes of the network may need to coordinate with other nodes to carry out some 
joint action HI, ||4|. To accomplish this coordination, the nodes could in general talk back and forth with 
each other Given capacity constraints on the links between the nodes, the goal may be to minimize the 
amount of communication needed to accompUsh the coordination. This is not a traditional communication 
problem. There is no explicit message to be transmitted, and any node or set of nodes involved in the 
coordination does not necessarily have to figure out the task to be performed by the other nodes. It 
is valuable to develop a general framework that includes both the traditional demands of transmission 
of (possibly correlated) sources over networks, and coordination demands. Such a model has started to 
evolve in the recent literature |@], ||5], see also [7|. Our formulation can be understood as a natural 
extension of these works. 
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Roughly speaking, we assume that the m nodes of the network are observing i.i.d. repetitions of X^^\ 
X^'^\..., distributed according to q{x'--^\ a^^"*)). Given a joint distribution q{x^^\ x^"^\y'^^\ y^™^), 
the final goal of the i*^ node is to compute the i.i.d. copies of Y^^^ after the communication over the 
network where X^^^\ X^'^\..., X^"^\Y^^\ Y^'^\..., F^™) are jointly distributed according to 
q{x^^\ ...,x^^\y^^\ ...,y^"^^). To do this, the nodes can exchange messages over the network at rates 
not exceeding the capacity constraints of the links. We assume that there is a directed edge from the 
i*'^ node to the j^^ node with the rate constraint Rij for 1 < i,j < m. We say that the network with 
the set of rates Ri j for 1 < i,j < m is admissible if the i*'^ node is able to create the i.i.d. copies of 
within a vanishing total variation distance. All (or a subset) of the nodes of the network may have 
access to common randomness at a certain rate in a given application. This depends on the possibility of a 
parallel resource that provides common randomness to the nodes at some constant rate. For simplicity we 
assume that no common randomness is provided to the nodes, although this assumption is not in principle 
necessary and can be further explored. On the other hand, private randomization at the individual nodes 
can very well be feasible in a practical network. Thus, we provide the nodes with such ability throughout 
this paper. 

Note that the traditional problem of communication of messages over a network can be thought of as an 
special case of the problem defined above. One just needs to suitably choose q{x^^\ x^™^ , y^^^ , y^™^) 
where the i.i.d. copies X^*) represent the information initially available to the i*^ node, and the i.i.d. 
copies y(*) represent the desired information of the i^^ node after the communication. Therefore the 
advantage of network coding over pure routing, or the insufficiency of Unear codes are of relevance 
here as well. Lastly we note that there are some similarities between the problem of "communication 
complexity" and this problem when y(*)s are functions of (X^^^, X^*")). 

The simplest model to consider is the network with two nodes. This special case is also interesting 
because one can partition the nodes of a large network into two sets, S and S'^, and treat the nodes on 
each side of the partition as a single supernode. Thus, any outer bound on the two-node problem results 
in an outer bound for arbitrary networks. Still, the problem is difficult to solve even in the special case 
of two nodes. In this paper, we restrict ourselves to the special case of two nodes where we prove new 
lower and upper bounds. Our results build upon and generalize some of the results in [4| and |(5|. 

This paper is organized as follows. In section JIJ we introduce the basic notation and definitions used 
in this paper. Section JII] contains the main results of the paper, and section |V] includes the proofs. 
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II. Definitions 

In this section we provide a rigorous definition for the model described in the introduction for the 
case of two nodes. A general model for the case of m terminals can be defined along the same lines, 
as sketched in the preceding section. This we omit, because we prove results only for the case of two 
nodes. 

Take an arbitrary q{x^^\ x^'^\y^^\y^'^^). Given an arbitrary natural number r, we say that (i?i2,^2i) 
is admissible with r interactive rounds of communication if for any positive e, there is a natural number 
no where the following holds for any n > uq: the two nodes observe n i.i.d. copies of X^^\X^'^\ i.e. 
xj^, x[^l. Assume that the first node is using the external randomness Mi, and the second node is using 
the external randomness M2. Random variables Mi, M2, and -^^^^^-'^j^^ must be mutually independent. 
Let Ci, C2, Cr- denote the interactive communication used in the code, and let C = (Ci, C2, Cr). 
We have H{Ci\Ci;i-iX[]lMi) = if i is odd, H{Ci\Ci:i-iX[2M2) = if i is even. We further have 

- V Hia) < Ru 
n ^-^ 

i:odd 

and 

- V H{Ci) < R21. 
n ^-^ 

i:even 

Following the communication, the first node creates Y^^^ , and the second node creates Y^^J such that the 
total variation distance between p{x[]l , xj^ , Y^^n > ^iS ) and the distribution of {x[]l , x[2 , y//j , Y^^ ) , 
constructed by taking n i.i.d copies of q{x^^\x^'^\y^^\y^'^^), is less than or equal to e. In other words 

E 

If r = 1, we have only one communication from the first node to the second node. 
For the special case of Y^^^ = X^^^ and X^^^ = constant and r = 1, one gets a problem that is a 
special case of the problem considered by Paul Cuff Q. 

III. Statement of the Results 

Consider the case of r rounds of interactive communication, meaning that the two nodes talk back and 
forth for r rounds. 
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A. Converse results 

• The case of r = 1 (i.e. R21 = 0): 

When y(^) = X^^\ X^'^^ = constant, one gets a problem that is a special case of Paul Cuff's result 
||5l . Cuff showed that in this case the minimum admissible R12 is equal to the Wyner common information 
M, i-e. 

inf /([/;X«y(2))^ 

y(2)-c/-x(i) 

We generalize this result for arbitrary q{x^^\x^'^\y^^\y^'^^): 

Theorem 1: Assume that r = 1, that is when R21 = 0. Then one must have Y^^^ — X^^^ — X^'^\ and 
furthermore (iii2,0) belongs to TZi defined as foUowsQ 

{(i?i2,0) G7^2 : 3p{u,x^^\x^^\y^^\y^^^) eTis.t. 

R12 > /(?7;x«y«y(2)|x(2))}, 

where 

T,^{ p(^.,x«,x(2),yW,y(2)): 

^(1) ^x(2)^ yd) ^ ym ^ ^(^(1) ^ ^(2) ^ yd) y(2) ) 

u-xW-x(^\ 

y{2)_[/x(2)-X«y«, 

• The case of a fixed r > 2: 

Theorem 2: Take an arbitrary q{x^^\x^'^\y^^\y^'^^). Then any admissible pair (-Ri2,^2i) must be- 
long to TZ2{r) defined as the convex closure of rate pairs (-^12,-^21) G such that there exists 

K/l,...,/r,a;«,x(2),y(l),y(2)) er2(r) such that 

R12 > /(X«;F|X(2)), 
R21 > /(X(2);F|X«), 
R12+R21 > /(XW;F|X(2)) + 

/(x(2);F|x«) + /(y«;y(2)|x«x(2)), 

'The authors would like to thank Mohammad Hossein Yassaee for pointing out a typo in the statement of this theorem. 
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where F = (Fi,F2,...,Fr) and T2{r) is the set of /2, y(i),y(2)) satisfying 

, , y (1) , F(2) ^ , , , y(2) ), 

i^i - - if ^ is odd, 

- Fi:j_iX(2) - if z is even, 

y(2) _Fx(2) -X(i)y(i), 
V 1 < ^ < r : 

• r/ze caje of no constraints on the value of r: 

Theorem 3: Take an arbitrary q{x^^\x^^\y^^\y^'^''). Then for any r, any admissible pair (-Ri2,-R2i) 
must belong to TZ^ defined as the convex closure of 

{(i^l2,i^2l)e7^2 : 3p(n,X«,x(2),y(l),y(2)) e S.t. 

Ri2 > /(X«;C/|X(2))^ 

R2I > /(X(2);C/|X«), 

R12 + R21 > /(xW;?7|X(2)) + /(x(2);i7|xW) 
+/(y(i);y(2)|x(i)x(2))}, 

where T3 is the set of p{u,x^^\x^'^\y^^\y^'^^) satisfying 



X«,X(2),y(l),y(2) ^ g(x«,x(2),ya),y(2)), 

y(i) -x(2)y(2), 

y(2)_[/x(2)-X«y«, 
|W| < |Af«||yi)||A'(2)||3;(2)|^i_ 
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IV. ACHIEVABILITY RESULTS 

Theorem 4: Take an arbitrary rr^^), y(2)) such that X^^) = X^'^\ and let X = X^^) = X^^). 

Assume that random variables Fi,F2, ■■■,Fr are jointly distributed with X,Y^^\Y^'^'> such that 

y(i) _ - 
Furthermore assume that positive reals R'^, R'2, ...,R'^ satisfy the following: 

VI < s < r : E-=i«: > I{Y^'\Y^^^;Fi,...,F,\X) 

Then the rate pair {J2odd i^'i^Yjeven i^i) achievable with two rounds of communication, i.e. with 
r = 2. 

Discussion: The above result is based on a generalization of Lemma 6.1 of [5|, which itself is a 
consequence of the resolvability work of Han and Verdu in [6 | and Wyner 

Lemma 6.1 of /J"/: For any discrete distribution p{f,v) and each n, let C(n) = Fi:„(t)^^^ be a 
"codebook" of sequences each independently drawn according to Y[k=iPFifk)- For ^ fixed codebook, 
define the distribution 



Qiv,..^) = 2-"^ Yl n Pv\F{vk\fkit)). 



i=l k=l 



Then if > I{V; F), 



lim E 

n— 00 



0, 



Q{vi:n) - n 
k=l 

where the expectation is with respect to the randomly constructed codebooks C{n). 

We now present the generalization of the above lemma that we use. To get back the above lemma, 
simply set r = 1 and X to be constant. Note that Theorem |4] can be proved by taking V to be Y^^\ Y^"^^ 
in the following lemma. 

Generalization of the lemma: Take some arbitrary discrete joint distribution p{fi, f2, fs, fr,v, x). 
For each n and every sequence xi-n G Af", we randomly construct a "codebook" C(xi:„) of size 
2nR\+nR'2+...+nR',. foUows: generate 2"^i sequences of length n independently, with each sequence 
distributed according to HiLi Let us label these sequences by /i,i:n(ii) for 1 < ti < 2"^i. The 

sequences fi.i:n{ti) (for 1 < ti < 2"^i) will be the cloud centers for the sequences generated in the next 
step. For every 1 < ti < 2"^i , generate 2"^^ sequences of length n using the conditional distribution of 
n"=i q{f2i\fii{h),Xi). Let us label these sequences by f2,i:n{tiM) for 1 < ti < 2"-^^!, 1 < ^2 < 2"-^^^. 
The sequences /2,i:n(ii5 ^2) (for 1 < ti < 2"^i , 1 < t2 < 2^^'^) will be the cloud centers for the sequences 
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generated in the next step. We continue this for r steps. In the r-th step we generate fr,i:n(ti,t2,ts, ...,tr 
for l<ti< 2"^'i, 1 <t2< 2"^s 1 < t3 < 2"^^ 1 < tr- < 2"^^. 
For a fixed codebook, define the distribution 

tl=lt2 = l t^ = l 

n 

/2fc(il,i2), frkitl,t2, ...,tr),Xk). 



k=l 



Then if 



Vl<s<r: Y.Ul^^>IiV■,Fl,...,Fs\X), 



we have 



lim E 

n—oo 



0, 



Qivi:n,Xl:n) - Y[PV,x{Vk, Xk) 

k=l 

where the expectation is with respect to the randomly constructed codebooks C{xi-n) and i.i.d. sequences 

Remark 1: For the case of X = X^^^ = X^'^^ and r = 1 the above theorem implies that the rate (i?i2, 0) 
is achievable if R12 > inf u-YW-ux-y-^^ /(y(^), y(^); J7|X). Compare this with the converse given in 
Theorem[T]which shows that any achievable (-R12, 0) must satisfy R12 > inijj.YW-jjx-Y'^^ 

Remark 2: The above theorem can be extended to scenarios where there are functions ki and /c2 
satisfying K = ki{XW) = k2{X(^^) and - - y(i)y(2). in this case the nodes can create K 

and work with it to reconstruct i.i.d copies of Y^^^ and y^^^ within an arbitrarily small total variational 
distance from the i.i.d. scenario. The constructed outputs together with X^^^X^"^^ will also have small 
total variational distance from the i.i.d scenario. 

Theorem 5: Take some arbitrarily distributed with X^^), y(i), y(2). Assume that the 

rate pair {R12, R21) is achievable for the problem of q{x^^\ x^'^\ z^^\ z^"^^) with ri rounds of communi- 
cation, and the rate pair (^'121^21) achievable for the problem of q{x^^^ z^^\ x^"^^ z^'^\y^^\y^'^^) with 
r2 rounds of communication. Then the rate pair {R12 + R!i2,R2i + ^21) is achievable for the problem 
of q{x^'^\x^'^\y^^\y^'^'>) with ri + r2 rounds of communication. 

Observation 1: If i?(y(i)|X(2)) = and F(y(2)|x(i)) = 0, then the set of (i?i2,i?2i) satisfying 
R21 > i?(y(^)|X(i)), R12 > i?(y(2)|x(2)) is achievable using Slepian-Wolf binning with two rounds of 
communication, i.e. with r = 2. Small probability of block recovery error implies a small total variational 
distance. 
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Corollary 1: The above observation together with Theorems 5] and |5] imply an achievable rate region 
for the general problem of q{x^^\ x^'^\y^^\y^'^^). The two nodes can first exchange X^^^ and X^'^\ by 
usmg up rates and One can then apply Theorem g] for the reconstruction 

of and y(2) 

V. Proofs 

Proof of The oremU} Take an arbitrary q{x^^\ x^'^\ y^^\ y^"^^). Since (-R12, 0) is admissible with one- 
way communication from the first node to the second node, for any positive e, a code of length n exists 
such that the communication rates are less than or equal to {Ru, 0) and the total variation distance between 
the overall reconstruction and the distribution of n i.i.d copies of {X^^\ X^'^\Y^^\Y^'^'^) is less than or 
equal to e. In other words for some natural number n, the two nodes observe n i.i.d. copies 
i.e. x[^l^,x[^l^ respectively. Assume that the first node is using the external randomness Mi, and the 
second node is using the external randomness M2. Mi, M2 and x[^lx[^^ must be mutually independent. 
Let C denote the message sent from the first node to the second node. We have H{C\x[^IMi) = and 

-HiC) < Ri2. 
n 

Following the communication, the first node creates Y^^^ , and the second node creates Y^^^ such that the 
total variation distance between p{x[]1,x[^1,yI2 , Y^^) and the distribution of (^{2, x[% y}^^ , Y^^), 
constructed by taking n i.i.d copies of q{x^^\x^'^\y^^\y^'^^), is less than or equal to e. 

The proof generalizes the one given by Paul Cuff in |)5l|. Take a random variable J uniform on 
{1, 2, 3, n} and independent of all the above mentioned random variables. Let 

u = jcx^l.xfl.^^, 

Note that the total variation distance between the joint distribution of {X^^\ X^'^\Y^^\Y^'^^) and that 
of (X(i),X(2),y(i),y{2)) distributed according to q{x^'^^ 

^ 2;(2) y(2)^ is less than or equal to e. 
This is true since for any I < j < n, the total variation distance between the joint distribution of 
{x!j^\xp,Yj^\Yj'^^) and the joint distribution of y^.^^\ y^.^^^) is less than or equal to e. 

We further have 

y{2) _ ^^(2) _x(^)y(^) 
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Since 

= I{Y}^;Xi]lY}2M,\CM2X[l) = (1) 
I(Yl^;iMr,xi}lY{}^M^\Cx[l) - /(M2;Xi2ySMi|CxS) > 
I{YlSMr,x[}lYl^,M^\Cxi2) - I{M2; x[}IyI^ M.Clx'i^l) = (2) 

/(yf x]^)y/^)|cxg_ix]5,^„xf ), V 1 < J < n, 

where equations [J] and |2] hold because ^(y/^^^ICMaXj^) = 0, and F(Cy//j|MiX{2) = 0. Next, note 
that 

The first term ^^(xf^^'SnSl^iS) is equal to -ii/(xS) + ^i^(X;2ySySxi2). Using Csiszar- 

Komer inequality [3 . Lemma 2.7] on }^Hix[}lYl}^Y}^x[^l), we can bound this term, i.e. I^S)' 

from below by 

— LfffX^'^^) _l_ i fffx*'"'" ''y y*'^'' A"''^''') -I- —]nff( e_ _ 'i 

n V^l:n/ ' n v l:n l:n l:n l:nJ ' n ^ov (| A"'!) 1 1 A'f^) | 1 13;(2) |)n ' 

= /7(xWyWy(2)|x(2)) + ^ ioff( '-^ — ^ — ) 

>ff(xa)ya)y(^)|x(^)) + dog( ,^,,,,,^,,-^,,,,^,^ 

where K(e) = — elog( |;|^(i)||;y(2)j|y(i)||y(2)| )• Here we used the fact (discussed above) that the total variation 
distance between the joint distribution of (X(i),X(2),y(i),y(2)) and that of (X^^), X^^), y(i), y(2)) is 
less than or equal to e. We bound the second term, i.e. —^H{x'^I^yI^^^^Y^^^\x'^I^C), from below as 
follows: 

-^T:,=iH{xf^Yinp\xfx^^l,xfl^^^^^^ 

Therefore 

i?12 > /(X(l)y(l)y(2); - 2K(e) 

Hence, (i?i2,0) belongs to TZ\ defined as follows: 

{(i?i2,0) G7^2 : 3p(^.,x«,x(2),y<i),y(2))erf s.t. 

^12 > /(c/;x«y«y(2)|^(2))_ 2^(^)1^ 
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where 

n 4 {p(n,X«,x(2),yW,y(2)) : _^(^(l)^^(2)^^l)^^2))||^ < ^ 

y(2) _ uxi"^) - 

since a cardinality bound on \U\ can be imposed using the generaUzed Caratheodory theorem of Fenchel. 

The proof will be done by noting that ne>o^i — ^i- order to show the latter, note that IZi C 
ne>o^i t'y definition. In order to show ne>o^i ^^^^ ^ sequence of ei, e2,.-- where 

limj_>oo = 0. Take a point in ni>i^i'- Corresponding to this point are pi{u,x^^\x'^'^\y^^\y^'^^) 
in Tl\ Since we have cardinality bounds on the alphabet of the random variables involved, one can think 
of these probability distributions as vectors in the probability simplex of T^l^ll-^'^'ll^^'^'ll'^'^'ll^^'^'l. Since 
the probability simplex is a compact set (when viewed as a subset of the Euclidean space), there must 
exist a subsequence 11,12, ■■■ where the sequence for /c = 1, 2, 3, ... converges 

to some p*(n, x^-*^), x^^), y^-*^), y^^^). p*(n, x'^'^\x^'^\^f^\y'^'^'>) must belong to Ti using the fact that mutual 
infomiation function and the total variational distance are continuous in the underlying joint distribution. 
One can further observe that the point in ni>i ^1 ^^^^ we started with, also belongs to TZi for the choice 
of p* {u,x^^\x'^'^\ip-\y'^'^'^) since limj_j.oo 2K(ej) is zero. ■ 
Proof of Theorem^ Take an arbitrary q{x^^\x^'^\y^'^\y'^'^'^). Since (i?i2,^2i) is admissible with r 
rounds of communication, for any positive e, a code of length n exists such that the communication rates 
are less than or equal to (i?i2,^2i) and the total variation distance between the overall reconstruction 
and the distribution of n i.i.d copies of (X(i),X(2),y(i),y(2)) is less than or equal to e. In other 
words for some natural number n, the two nodes observe n i.i.d. copies of i.e. 
respectively. Assume that the first node is using the external randomness Mi , and the second node is using 
the external randomness M2. Mi, M2 and xj^xj^^ must be mutually independent. Let Ci,C2,...,Cr 
denote the interactive communication used in the code. We have H{Ci\Ci-i-iX'^l^Mi) = if z is odd, 
H{Ci\Ci.,i-iX^^lM2) = if Hs even. We further have 

- V H{Ci) < R12 
n ^-^ 

i:odd 

and 

- V H{Ci) < R21. 
n ^-^ 

i:even 
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Following the communication, the first node creates Y^^^ , and the second node creates Y^^^ such that the 
total variation distance between x[%y{2 , y}2) and the distiibution of {x[]l, x{f^, y}^ , Y^^), 

constructed by taking n i.i.d copies of q{x^^\x^'^\y^^\y'^'^^), is less than or equal to e. 

Take a random variable J uniform on {1, 2, 3, ra} and independent of all the above mentioned 
random variables. Let 

^ - -^^LJ-l^J+ltn' 

F, = a (for I = 1,2,3,..., r)andJ^ = {Fi, F2, F,.), 
= = 42)^^(1) ^ y(l)^y(2) ^ y(2) 

The following statements hold: 

1) /(y(i);X(2)y(2)|x(i)^Z) = since, for any 1 < i < n, we have: 

= I{X^lM,;XSlM,\X^llx[^l,) = /(xj^MiFi; > 
I{x[}lMv,x[^lM2\X^2xi^l,F,) = I{X[^^^^^ > 
/(xi2Mi;XSM2|xj2xg_,Fi:2) > • • • > /(xj^Mi; X^M^lxj^xgL,:^) > 

r^y>{l). 3^(2)<>(2)|3^(l)^(2) "^^ .^>(1) (2)^>(2) , ^^(1) ^(1) „(2) 
-'l^l:n'^l:n-'^l:nl^j:n^l:j-l-^ J ^ -'l-'^j ' ^j+l:n^l:j-l^ )■ 

Therefore /(y(^); X^^^y^^^jX^^^^Z) = 0. Note that this equation implies that for any value of 
Z = z, I(y(i);X(2)y(2)|x(i);^,Z = z) =0. 

2) Similarly one can show that I{Y^'^'> ; X'^'^'>Y'^'^'>\X^'^'>^ Z) = 0. Note that this equation imphes that 
for any value of Z = z, 7(y(2);X(i)y(i)|X(2);^, Z = z) = 0. 

3) = if z is odd, since following the begimung steps of the proof for the 
first statement, one can show that for any 1 < j < n the following inequality holds: 

> /(xi2Mi;XSM2|4n^S-i^i-i) 

Next, note that 

m;Xf\X^lx^j_,F^:,^,) = I{F,; xf\X^^ X^^,^^X^j_,F,:,_^. 

Therefore I{Fi;X^^^\ Fi:i-iX^^^ Z) = if i is odd. Note that this equation imphes that for any 
value of Z = z, I{Fi; X^'^^\Fi.,i^iX^^\ Z = z) = if Hs odd. 

4) Similarly one can show that = if i is even. Note that this equation 
implies that for any value of Z = z, I{Fi; X^^^ Z = z) = if i is even. 
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These statements imply that conditioned on any Z = z, the conditional distribution 

p(/l,/2,...,/r,X«,x(2),y(l),y(2)|^) 

satisfies following Markov chains equations: 

Fi - Fi,i_iX(i) - if i is odd, 
Fi - - if i is even 

_ - X(2)i>(2) 

y(2) _ - 

Let (i^ denote the total variation distance between -piX^^^ = x^^^X^^) = x^^^jF^^^ = y(^),y(2) = 
y(2)|z = z) and = x(i),X(2) = = y(i),y(2) = y(2))_ appendix S we show that 

In appendix E we show that the triple (i?i2, ^21, ^12 + ^21 - /(r^^); r(^)|X(i)x(2)) + 3K(e)) is 
coordinate by coordinate greater than or equal to 

Z,piz)(^Iif;XW\X(^),Z = z),Ii'f;X(^)\Xm,Z = z), 

I{f;XW\X(^\Z = z)+l(f;X(^^\X(^\Z = z)^ 

where K(e) = -e log( |^(i)||^„)J|y(i)||y(,)| ) and /(y^^); is evaluated assuming that 

X(i),X(2),y(i),y(2) have the joint distribution of x^^), y(2)). 

Therefore the four-tuple (i?i2, i?2i, -R12 + i?2i - /(F^^^ F^^^IX^^^X^^)) + 3K(e),e) is coordinate by 
coordinate greater than or equal to 

EzPiz)(ii^;xW\x('),z = z),i(f;x(^)\xm,z = z), 

,Z = z)+ i(F; , Z = z), 4) • 

The Caratheodory theorem implies that the four tuple {Ru, -R21, -R12 + ^21 - /(F^^^; y(2)|X(i)x(2)) + 
3K(e), e) is coordinate by coordinate greater than or equal to 

Ei=i:5 r^ (/(i^; |X(2) , Z = z,) , I ; X i^) \XW ,Z = Z,), 

l(f;X(')\X(^\Z = Zi)+I(F;X(^^\XW,Z = 
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for some values of zi, Z2,..., z^ and non-negative ri, r2,...,r5 where Yli=i-5 ''i = 1- Since for i = 1, 5, 
we have ridzi < e, if dz^ > \/e, Vi will be less than or equal to s/e. Let t = Xlrd, <ye''«- that 
1 - S^/e < t < 1. We then have: 



lif; , z = + /(^; x(^) , z = zO, 4.) . 



is coordinate by coordinate greater than or equal to 



Z = z,) + Z = z,), 4.) 

Therefore (i?i2, ^21, i?i2 + i?2i - /(F^^); + 3^^^^) coordinate by coordinate greater 
than or equal to 



, Z = + , Z = Zi)^ 



Therefore any admissible pair {R12, R21) must belong to 'R-2{r) defined as the convex closure of 

{(i^l2,i^2l)e7^2 : 3p(/i,/2,...,/„xW,x(2),^l),y(2)) e r|(r), 

, , y (1) , y (2)) ^ , ^(2) , y(i) , y(2)) s.t. 

i?i2 > (l-5V^)/(X«;i^|X(2)), 

R21 > (i-5Vi)/(x(2);:^|xa)) 

R12 + R21 > (l-5Vi)/(^^'^;^|x(2)) + (l-5Vi)7(x(2);:^|X«) + 
+/(y(i);y(2)|xa)x(2))-3/«(e)}, 
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where 

Fi - - if i is odd, 

Fi - - if i is even 

y(i) _ 'fx^^^ - x(2)y(2) 
V 1 < i < r : 

< I-F.-1II-F.-2I • • • |-Fi||A'W||3;«||Af(2)||3;(2)| + 1}. 

since cardinality bounds on can be imposed using the generalized Caratheodory theorem of Fenchel. 
This is argued in appendixjC] We can then continue as in the proof of theorem[T]to show that ne>o ^K'^) — 
lZ2{r), and hence (-Ri2,-R2i) must belong to 'R\{r). ■ 
Proof of Theorem\^ It suffices to show that Ur>i^2(?') C 7^3. Take some r and some point 
{Ri2,R2i) G Tl2{r)- Corresponding to this point is some p{fi, f2, fr,x^'^\x^'^\y^^\y^'^'^) G 72(r). Let 
U = Fi:r be jointly distributed with X^^), X^^) , y(i), y(2) according to p{{fi, /a, fr), a^^^^ , x^^), y(i), y(2)). 
Note that /(X^^) ; X^^) |[/) < I(x(i);X(2)) since 

I(XW;X(2)) =/(x(i)Fi;X(2)) > /(xW;X(2)|Fi) = /(X«;X(2)F2|i^i) > 
/(X(i);X(2)|Fi:2) > ••• >/(X(i);X(2)|Fi.,) =/(X«;X(2)|[/), 

where we have used the fact that for any p{fi, f2, fr, G T2{r), the following 

Markov chain equations hold: 

Fi - - X(2) if i is odd, 

Fi - - if i is even 

One can verify that p{u, satisfies the required properties for being in T3, except 

the cardinality bound. Similar to what was done at the end of the proof of theorem |2] one can use the 
generalized Caratheodory theorem of Fenchel to impose the desired cardinality bound \U\. ■ 
Proof of Theorem ^ It suffices to prove the generalized lemma stated in the discussion following 
the statement of the theorem. Here we build upon the ideas Paul Cuff uses to prove the direct part for 
the problem he considers. 

We denote the induced joint distribution on (xi:„, y^^^, by P{xi-n, Vi^l, y^n)- We use the capital 
letter P to indicate that this probability is a random variable due to the random generation of the codebook. 
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The induced joint distribution on (a^iin; y^n; y^n) is equal to: 

(n n 

l<ti<2""i lor i=l,2,..,r »=1 

flqiyl''^\Fl'\ti),Fl'\hM,---,Ft\h,t2,h,...,tr),Xi^^ 
i=i ^ 

To prove the theorem, it suffices to show that the expected value of the total variance between the random 

variable P{xi;n,y[^n,y^^n) Yli=i'li^i^yi^\yi^^) converges to zero as n converges infinity, i.e. 

n 

lim Yl E|P(xi:„,yS,yS) - E «(^- ^^'^ = 0" 



(1) (2) 
^l:n)J/i:n)f l:n 



i=l 



Using the Unearity of expectation, we show that E[P(xi:„, y^l^, y^l)] = HILi li^i^ ■>yf^) follows: 

E[P(xi:„,yS,yS)] = 

- n 



n c/(x,)2-"5:L. n'. J] q^yf\FP (1), (1,1),..., (1, 1, 1, 1), : 
i=l i=l 

n 

n qiy?^\F^'^ (1), (1, 1), Ft^ (1, 1, 1, 1), X, 
1=1 

n r " 

n qixi)E n ^ il),F['^ {l,l),...,Ft\l,l,l,...,l),Xi) 

i=l 

n (1), F^ (1, 1), F^^ (1, 1, 1, 1), X,) 

flq{fi'\l),fP{l,l),...jt\hhl,...,l)\xiy 

i=l 

U^{yl'\yr^\f^^{l)jr^{l,l),...jP{l,l,l,...,l),: 



1=1 '-i=i 

n 



i=l 



j=l j=l i=l 

Take some arbitrary e > 0. We first take care of the sum of the total variance over those of non-typical 
sequences Te^''\ Using the inequality E[|X — E[X]|] < 2E[X] for non-negative random variables, we 
can write 

n 



i=l 
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5^ 2EP(xi:„,yS,yS) = 

n 

It remains to bound 

n 

Let us define A(a;i:n, ^12' ^i-n) A(a;i:n, y^ll, as follows: 

Pi{xi:„,yi]l,yifj = 



(n n 
]Jq{x^)2--^^-^^^]Jq{yl'\yl'^\F^^'\t,),Fl'\h,t2),...,Fl^^ 

tor i=l,2,..,r '=1 ^=1 



l<ti<2 



(n n 
J{q{x,)2--^U^'^J{q{yf\yf^\Fi'\t,),Fi^\t,M),...,Fi'^^ 

""i lor j=l,2,..,r '=1 ^=1 

Using tlie triangle inequality and the fact that -P(xi:„, y^^' ^1:1) = A (^^i^n, yLn)+-P2(a;i:n, y^-n' Vil) 



we can wnte 



Therefore 



E|p(a;i:„,yS,yS) -EP(xi:n,yS,yS)| < 

E|P2(xi:„, yS> - EP2(xi:„, yS> y?l) I • 

n 

E|A(xi:„,yS>yS) -EA(xi:„,yS>yS)|- 



DRAFT 



17 



Let us begin with the second term 



J2 E|P2(xi:n,yS,yi?^) -EP2(xi:n,yS,yif^)| < 

J2 2EA(xi:n,yS>2/S)< 



2 Yl ng(xi)E(ng(yf\yf)|i^«(ti),i^(')(ti,t2),...^ 

E 

n n 

n «(^^) n ^(fi'^ /i'^ *2), (il , *2, t3, «r-)|^i) 

1=1 i=l 

n X 
n yf ^ (il), /f («1, i2), (tl, t2, t3, tr), X,) < 26. 

i=l ^ 
We will work out the first term now 

J2 E|A(xi:n,yS,yS) -EA(xi:n,yS,yS)| < 

E VE(A(xi:n,yiiyS) -EA(xi:„,yS,yS))' = 

5; Vvar(A(xi:„,yS,yS)). 
We now compute the variance of Pi{xi:n, Ui^^, y^l) using the formula Var{Ki + K2 + ... + K^) 

Var(A(xi:„,yS>yS)) = 



E Cov\ 

l<ti<2'''^i ,l<ti<2""-i for i=l,2,..,r 
n n 

i=l i=l 

i[iySlJ'lFi^^ih),...,Fii{t,,t,^^^^ 
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i=l i=l 

The number of sequences {ti,t2, ■■■,tr) and {ii,t2,is, ...,ir) that match in the first s coordinates but 
differ in the (s + l)-th coordinate, i.e. ti = i\,t2 = ?2, ■■■,ts = is,ts+i ^ ig+i is less than or equal to 
2" 5];j=i ^« 2^" ^*=8+i ^* . Given a fixed s, the covariance of any two terms corresponding to {ti,t2, ■■■,tr) 
and (ti, ^2, ^3, ir) are the same since from the (s + l)-th stage onwards the F sequences will fall into 
different cloud centers and the actual value of the indices ts+2,ts+3, ...,tr, 4+2, 4+3, ir have no effect 
on the joint distribution of the two terms whose covariance is being computed. Therefore the variance 
of Pi(xi 

■ni y^n) Vvn) ^'^^ bounded from above by the summation of s from and r of 

2"E|=iR^22"5:r=,+i«JC'oi; 

n n 
]Jq{x,)2--^^--^^^]Jqiyl'\yP\F['\l),FPil^),...,Ft^ 

i=l 1=1 

, and 

n n 

s terms s terms 

^[{y^Zy?lF[]l{\),...,Fi'\w,...A),Fl^^'\i^^ 

s terms s terms 

The above term is equal to the summation of s from and r of 

n 

1=1 

n 
i=l 

1 [{yil (i> h 1, 1), (1, 1, 1, 1), i^S (1, i> 1, i)> ^i:n) G ri") ] 

, and 

n 

n i(yl'^'y?^ i^^'^(i)' ^^^^ 1' 1' 1)' ^^^^'^ (M^, 2), i^^'-) (M^, 2, 2), X,) 

s terms s terms 

1 [(yS, yil (1), ^^'^ (1, 1, 1, 1), ^^'^'^ (1^^^ 

s terms s terms 
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When s = 0, the covariance is exactly zero. For s > 0, we can use the inequality Cov{X, Y) < Ei{XY) 
for non-negative random variables to bound the covariance term 

Gov 

n 

1=1 

, and 

n 



]Jq{yi'\yP\F^^'\l),...,Fl'\l,l^,...^),F;'^'\lAA,---h2),Fl''\ 

—1 ' V ' 

'~ s terms s terms 

1 [(yS, yil (1), ^^^^ (1, 1, 1, 1), ^^^^'^ (1:^^ 

s terms s terms 



from above by 



^ i=i 

n 

]Jq{yi'\y?\F^^'\l),...,F^^'\l,lA,...A),F^^'^'\lA^^ 

s terms s terms 



s terms s terms 

We can bound this further from above by 

^(fl<l{yi'\yP\F^'\l),...,F^^'\l^A,...A),F^^'^'\l^^^^^ 

^1=1 

n 

'~ s terms s terms 

Let us compute this expectation by first conditioning it on F^^\\), F^^^ (1, 1, 1, 1): 

E(U'iiyl''M"¥l'\i),---,F,^'\iAA,...A),Fl'^^^ 



n liVi ' yrl^r (1), (1,1,1,-, 1), ^^'^'^ (M^, 2), f}^^ (M^, 2, 2), X,). 

s terms s terms 
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E ( n \ (1), f/*-) (1, 1, 1, 1), 

n 

ns s tenns 



s terms 

(1) .,(2) 



1 Ml y^l F^l (1), (1, 1, 1, 1), G r/") 

E n ' ^ 1^^'^ (1)' i^^'^ (1, 1, 1, 1), ^0 

Using the fact that when 
are jointly typical, the term 

n 



i=l 



is less than or equal to 2 "(^(^^ |f(i\...,f('*>,x) e)^ further bound the covariance term from 

above by 

(n 
i=i 

n 

1=1 

Since y^}^ is jointly typical, nr=i ^(2/1^^ 2/1^^ less than or equal to 2~"'(^(^'^''^^^'l^)~"^). 

Therefore the covariance term is less than or equal to 2-"(^«''''^/'V<^',-,^^=',^)-^)-"(J^(y^^>,^<^'l^)-^). 
Thus the variance of P\{xi.n, y^^l y^l) is less than or equal to the summation of s from and r of 

JJg(^.)22-nEU R',2-n{H{Yl'\Yl'^\F<-'\...,F(^\X)-e)-n{H{Y('\Y(^^\X)-e)_ 
i=l 
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Since xi^n is typical, HILi li^i) 1^^^ t^an or equal to 2~2n(^^(-''^)-e) xhus, the variance of A(a^i:n, 2/^1' ^i^n) 
is less than or equal to the summation of s from and r of 

Remember that the total variance was shown to be bounded from above by 

Vvar(P(xi:„,yiiyS)). 

Our upper bound on the variance of Pi{xi-j-i,y]^n,y^n) does not depend on the particular realization of 
xi:n, Vi^n, 2/^1- Sincc the number of jointly typical sequences of {xi-n, Ui^n, Ui'-ld bounded from above 
by 2"(^(^'^*^''^*^')+'^), the total variance will be less than or equal to square root of the summation of 
s from and r of 

22n(/f (X,y(i\y )+e)-2n(^^(X)-e)-nELl^;-"(^^(^/'^■*^/'''l''^*'^•••.■P'°^^)-f)-"('f^(^''^■^*''l'^)-^) = 

2-n{Y,U^R'-I{Y^^^\Y'--'^;F'^^\...,F^='>\X)-7e) 

This term converges to zero for small values of e since 'Yli=iR'i > I{Y^'^\Y^'^^ ; F^^\ F^^^\X). ■ 
Proof of Theorem |5} Take some e > 0. One can then find a natural number no such that for 
any n > no, there are (n,e) codes for both the problems of generating q{z^^\ z^'^'>\x^^\x^'^^) and 
q{y^^\ y^"^^ \ x^^^ z^^\ x^'^^ z^"^^) with the given rounds of communication and rate pair constraints achieving 
total variation distances less than or equal to e. We claim that concatenating these codes would give us a 
(n,2e) code for the problem of generating q{y^^\y^'^^\x^^\x^'^^) with ri + r2 rounds of communication 
and the rate pair {R12 + Ri2, R21 + ^21)- First of all, one can run the concatenated code with ri + r2 
rounds of communication. If ri is even, then it will be the first party's turn to start the simulation of the 
second code. If ri is odd, one can interpret the last communication of the first party in the first code and 
the first communication of the first party in the second code as a single round of communication. This 
adds up to ri + r2 — 1 rounds of communication for the concatenated code which is even better. The rate 
of the concatenated code is clearly {R12 + R'u^ ^21 + ^21)- remains to check the total variation 

distance in the concatenated code. We know that 
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E 



,(1) .(2) (1) (2) 
'*'l:n''*'l:n''*l:n''*l:n'£'l:n'f l:n 



_ y(2) _ ,,(2) 1^(1) _ (1) ^(2) _ (2) ^(1) _ (1) ^(2) _ (2), 



„^y(l)_Jl) y(2)_,,(2)| (1)_ (1) ^(2) _ (2) _ Jl) ^(2)_ (2)x\ 

_ v^(2) _ ^(2) _ Jl) 7(2) _ J2)x < ^ 

^^V^l:n — -^lin' ^l:n ~ -^Irn' ^l:n ~ ^l:n' ^l:n ~ ^lin/* — ^• 

Multiply the first equation by 

_ ,,(1) y(2) _ ,,(2) I (1) _ (1) (2) _ (2) ^(1) _ (1) ^(2) _ (2). 

and sum it up over all j/^!^, and y^^; then add this with the second equation above and use the triangle 
inequahty to conclude that 

„(2) ,(1) .(2) (1) (2) 
■^IrnJ'^lini^liTH-^lrn'yirn'yirn 

„^y(l) _ ,,(1) y(2) _ ,,(2) (1) _ (1) (2) _ (2) (1) _ (1) (2) _ (2) . , 
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Appendix A 



In this appendix, we show that YlzPi^)^^ < e. Take an arbitrary 1 < j < n, 



^ ^ Z^a-d) 0.(2) 
•^I:n5'*-l:n5i/l:n)'*'l:n 



(1) x^(2) 



„(2) 



_ ^(1) J^(2) _ ^(2) y(l) _ (1) y{2) _ ^(2) |^(1) _ ^V-J 



(1) v(2) 



(2) 



'(1) v(2) _^(2) 



> 



y 



(1) „(2) ,,(1) ,,(2) 
>J-3 jS/j Ji/j 



r(l) ^(2) ,,(1) ,,(2) 



PV^y.n ~ -^j-.n^^l-.j ~ -^lij' -"^lin ~ yi:n' ""^Irn ~ i/l:nl^l:j-l ~ -^l-.j-n ^j+l:n ~ -^j+l-.n 
P\^j:n ~ -^j-.n^^l-.j ~ ^ l:n ~ ill-.n^ ^ l:n ~ yi:nl"^l:j-l — "^j+l:n ~ ■'^j+l-.n, 



= ^x^.,,x^P^^-^ = ^i^-l'^+l:n = ^}+l:n) >< E.O 



(1) 



.(2) 



(1) (2) (1) (2) 
j '^3 'Vj 'Vj 



) y(2) _ (2)|^(1) _ (1) ^(2) _ (2) A 



-p(xj^) = xj.^) , xf = xf , rj-^^ = -(^) vC^^ - -"(2) I ^(^) - -(^) ^(2) - -(2) 



Since the above equation holds for any 1 < j < n, we get YlzPi-^)^^ < e. 



Appendix B 



In this appendix, we show that the triple [Ru, i?2i, i?i2 + R21 - I{Y(^^;Y('^^\X^^^X('^^) + 3«;(e)) is 
coordinate by coordinate greater than or equal to 

= ^),/(:^;x(2)|xW,z = z), 

X(i) |X(2) , Z = z) + /(:^; X(2) |X(1) , Z = j 



and /(y(i);y(2)|x(i)x(2)) is evaluated assuming that 



where ^(e) = log( ; 
X(i),X(2),y(i),y(2) have the joint distribution of g(x(i),x(2),y(i),y(2)). 
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Ri2 > }H{CuC^,C^,...) > ^H{Ci,Cs,C5,...\M2X[ll) = iF(Ci, C2, C3, ...|Af2^|fi) 

> ki0;xSl\M2xSl) ='^Yr,=,i(^-,xf\M,x^^lxflj = i(P-,xi^)\^^^^ 

= H{X(^^\M2Xfl^.^^X(^)Z) - HixW\M2Xfl^.^^X(^^zf) 
= - H{X(^')\M2Xfl^._^X(^'^zf) = /(M2Xi^|i^„^;X(i)|X(2)z) 

>l(f;XW\x^'^)Z) 

• Similarly we have: 

i?2i >/(^;X(2)|x«Z). 

• Note that 

R12 + R21 > ^H0\M2X[^1) + ^H0\M,xi2) > 

^I{d;XSl\M2X^l) + iiJ(-^|M2Xif)xi2) + l/(^;XS|MiXi2) + ^H0\M,X<i^lx^l) 

The first and the third term are respectively greater than or e qual to l(f;XW \x(^)z) and ^(2) [X^-^) Z) 
as shown above. For the second and the fourth term, we have: 

^H(^\M2xi^lx^l) + ^H0\M^X^lxSl) > 

I{M^-^■,M2'^\x[^lx[]l) 

since for any odd i, 

7(MiCi:i;M2Ci:i|xSxJ2) = /(MiCi:^; M2Ci:i_i|xSx{2) + /(MiCi:i;a|x{2xJ2^2Ci:i-i) 

= /(MiCi.i_i;M2Ci.i_i|X;f]x;2) + I{MiC\..i; a\x[2x[}lM2Ci..i.i) 
< /(MiCi:,_i; M2Ci:,_i|x}f)x;2) + H{Ci\x[^lxi]lM2Ci..i.i) 

= J(MiCi:,_i; M2Ci:,_i|xSxi2) + i?(Q|Xi'ixi2M2Ci:,_i) + (Q IX^Xf^Mid:,.!) . 

Similarly, for any even i, 

I{M,C,..f,M2C,..i\x[^lxSl) 
< /(MiCi:i_i; Af26\;.-iixif^xi2) + H{Ci\x'i:lx^^lM2Ci..i.i) + H iCi\xi2x[]lMiCi:i.i) . 
Further, /(Mi; Mal^if^xJ^) = 0. Hence 

^H0\M2X'^IX^I) + ii/(^|MiX}2x}2) > 
il(Mi^;M2^|xSxi2) 
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Since 1"/.^^ is created from Mi^ x[^l and Y^^^^ is created from 1^2^ x[^l, we have: 

^H0\M2XSIX[}1) + ii7(^|MiXi2xS) > 

Note that 

1 7-/0(1). v(2)|)(-(2)v^(1)n _ 
n-'-y^l-.n I l:n l^l;n^l:nJ — 

1 H-r^Xl) V(2) ^(2) 1 o-/ ^(2) 

Using the Csiszar-Komer inequaUty [3, Lemma 2.7], we can bound this expression from below by 

~n^0^i-.n^i-.n^il-^i-l) ~ ^^i^ll^lJi) + (|;f (i) | |;f (2) 1 13)(2) |)„ ) = 

— /'('"K*-"'"'' ■ y*'^'' I x'-^'* x''"'"'''! -I- ln?(' e_ _ 'i > 

n V l:n ' l:n I l:n l:n^ '^n (| A'(i) 1 1 A't^) | |3;(i) 1 13;(2) |)" — 

^^(nS;nSl^S^J2) - 34^) = /(y«;r(2)|x(2)x«) - 3.(6). 

Therefore 

R12 + R21 > + + y(2)|x{2) _ 3^^^) 

Appendix C 

We begin by imposing a cardinality bounds on | J^i | , then on 1 7^2 1 > and so on. Having imposed cardinality 
constraints on |^a,-2|> ■■■ and will we show that the cardinality of can be bounded from 

above by ■ ■ ■ |^i||Af«||yi)||A^(2)||y2)| + ^ 

In order to impose the cardinality bound on Fi, we fix p{f2, fr,x^^^ , a;(2) , , y^^^ |/i) and vary the 
marginal distribution p{fi). The equations /(F^; X(2) |Fi:i_iX(i)) = for odd i, I{Fi; X^^) |Fi:i_iX(2)) = 
for even i, I(Y(^')- X^^'^Y^^'^l'f X^) = o and /(y(2). j^(i)|;^^{2)) 

= hold irrespective of p{fi). We 
need to impose | | \y^'^^ \ jS^^^^ | -1 equations ensure that the joint distribution of p{x^^\ x^'^\y^^\y^'^^) 
does not change, one equation for the probability constraint — 1' ^^'^ '^^^ equation corre- 

sponding to the two terms ^|X(2)) and I{X('^'>;^\X^^')) (since we are using the generalized 

Caratheodory theorem of Fenchel). Therefore we get the cardinality bound of Wy^^'' \ | ^"(2) 1 13^(2) | _|_ i 
on In order to find cardinality bounds for \T2\, we fix 

p{h,h,...Jr,x('\x^'\yW,y('^\h) 
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and vary the marginal distribution p{f2)- The equations I{Fi;X^^'> = for odd i > 3, 

= for even i > 3, /(y«; X(2)y{2)|:^x(i)) = and = 
hold irrespective of p{f2)- Next, we impose | J"i||A'(i)||5'(^^||Af(2)||5;(2)| _ i equations ensure that 
the joint distribution of p{fi,x^^\x^'^\y^^\y^'^^) does not change. This implies that the equations 
/(Fi;X(2)|Fi:^_iX(i)) = for odd i < 3, /(F,; = for even i < 3 hold. This 

argument can be repeated for F3, F4, ... and so on. 
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